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ABSTRACT 



The cognitive levels of instruction of professors from the 
Pennsylvania Governor's School for the Agricultural Sciences (PGSAS) and the 
cognitive levels of thcpught among students were studied. The classes of 4 of 
16 PGSAS professors were selected for analysis, and researchers recorded the 
frequency of observable teacher behaviors from each level of Bloom's 
"Taxonomy" in 6-minute intervals. Professors also completed a questionnaire 
about teaching skills and knowledge of cognitive levels of teaching. The 
second target population for the study was 64 scholars at the PGSAS during 
the summer of 1998. These subjects completed a questionnaire for background 
information before completing think aloud protocols about their thought 
processes during class. Findings show that professors were generally teaching 
at lower cognitive levels, with the most common teaching behaviors being 
basic elicitation of facts, verbalizing from or about graphic 
representations, and making generalizations or conclusions. When professors 
did teach at higher cognitive levels, the most common behaviors were 
producing unique communication and divergent ideas, showing interactions and 
relationships, and applying abstract knowledge. Students, however, primarily 
thought "random nonsense thoughts" during lectures. The implications of these 
findings are discussed, and some recommendations are made for improved 
instruction. (Contains 2 tables and 24 references.) (SLD) 
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USING THINK- ALOUD PROTOCOLS TO ASSESS COGNITIVE LEVELS OF 
STUDENTS IN COLLEGE CLASSROOMS 



Introduction 

The need to have students graduate with a demonstrated capacity to think at the 
higher levels of Bloom’s Taxonomy is more urgent than ever (Newcomb, 1995). 
Recently, however, much concern exists that college and university students are not 
learning to their full potential. According to Whittington and Bowman (1994), several 
major national reports expressed the view that undergraduate education in general is 
incoherent and ineffective. The apparent foundation for these accusations is failure on the 
part of educators to challenge students to think. However, using Bloom’s Taxonomy 
(1956) as a basis for examining cognitive levels of thought, it is possible to study both 
professors and students to determine the validity or lack of validity of these assertions. 

Theoretical Framework 
Bloom’s Taxonomy of the Cognitive Domain 

Bloom’s Taxonomy of Educational Objectives: Cognitive Domain provides a 
useful framework for documenting the various cognitive levels at which the brain 
operates. This framework gives focus and direction to teachers who are looking to 
improve the quality of learning in their classrooms (Whittington and Bowman, 1994). 

Bloom’s (1956) six-step hierarchical system of thinking moves from knowledge 
which emphasizes recalling subject matter, to evaluation which entails making judgments 
(Table 1). Each level is reflected through cognitive activities. 

Review of Literature 



Think-aloud Protocols 



During the 1950’s, the cognitive revolution initiated a new era of thinking about 
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g fundamental questions about the human mind and by creating 



perspectives and tools to pursue the answer to those questions. Think-aloud protocols, 
the verbal reports produced by subjects who expressed their thoughts while engaging in 
some activity, has been one of the tools that allowed psychologists to explored previously 
inaccessible domains of cognitive processing (Kucan and Beck, 1997). Newell and 
Simon (1972) analyzed think-aloud protocols, and demonstrated how the pieces needed to 
fill-in the cognitive puzzle’s vast and empty interior. 



Higher-order Thinking 

Higher-order thinking is defined as application, analysis, synthesis and evaluation 
(Bloom et al., 1956). Thomas (1987) further defined higher-order thinking as the ability 
to think critically, make ethically and intellectually defensive decisions, and reason. 



Table 1 



A Synopsis of Bloom’s Hierarchy of Thought 



Cognitive Level 


Definition 


Activity 


Knowledge 


Recalling subject matter 


List, define, label, and match 


Comprehension 


Learners know information that has 
been communicated, but cannot apply 
in other situations 


Explain, rewrite, paraphrase, 
summarize, and give 
examples 


Application 


Learners apply information to different 
situations and learning tasks 


Compute, demonstrate, use, 
predict, discover, and solve 


Analysis 


Learners separate data into its 
component parts; these parts are 
differentiated and related based on 
their relationship 


Differentiate, discriminate, 
relate, diagram, and 
distinguish 


Synthesis 


Combines learned elements to create a 
new whole; working into pieces and 
elements, arranging so as to create 
new forms, patterns, or structures 


Create, compose, produce, and 
develop 


Evaluation 


Entails making judgment on the value 
of materials and methods for given 
purposes 


Justify, compare, contrast, 
evaluate, and interpret 



Note. McCormick, D. (1998). Adapted from Bloom et al., (1956). 



According to Thomas, a higher-order thinker asks questions that probe what is 
known, deducts possible outcomes of a particular situation using principles, and tests 
one’s own line of thinking and reasoning. Higher-order thinking requires the use of basic 
thinking skills such as knowledge recall, comprehension, and application, but analysis, 
synthesis, and evaluation are its primary cognitive requirements (Bloom, 1984). Since 
information is only useful when it can be applied and used for solving problems and 
making predictions (Underbakke et al., 1993), higher order thinking provides a 
foundation for effectively dealing with information (Halpem, 1984). 

However, compared with educated students of other nations, our students are 
falling behind academically (Sternberg, 1985) due to underdeveloped higher-order 
thinking skills. For U. S. students, performance on moderately complex and scientific 
tasks has not changed in almost a decade, and only a small number of students, merely 7 
% of 17 year olds, demonstrate such higher-level skills (Paul, 1992). 

Thus, a lack of correspondence exists between that which is needed to develop 
higher-order thinking, and that which is actually being offered to students. Lecture, the 
most commonly used instructional method in the university system, often only offers 
students the opportunity to develop higher-order thinking skills on their own (Bostick, 
1996). Most students, though, do not have the background in thinking skills 
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development or metacognitive analysis to develop their own thinking skills (Paul, 1992). 
Ericksen (1984) concluded that students will learn what they find interesting and only 
remember what they understand. 

Previous studies have stated that professors often concern themselves with the 
content of their lectures, but spend less time thinking about student performance and the 
cognitive level their instruction reached (Whittington and Newcomb, 1991). In reality, 
the necessary higher-order thinking skills needed by students can only be developed 
through a learning environment that consciously teaches thinking skills and provides 
opportunities for interaction (Thomas, 1987). 

However, a study by Cano and Newcomb (1990) showed that 60% of teacher 
instructional methods focused on knowledge and comprehension; 20% on application and 
analysis; and another 20% on synthesis and evaluation. Whittington (1995), in a study of 
30 professors from The Pennsylvania State University, found that nearly 80% of 
discourse in college classrooms was at lower cognitive levels. 



Engaging Thinking in Classrooms 

The outcome of a curriculum should be to engage students in thinking at higher 
cognitive levels (Whittington and Bowman, 1 994). Cano and Newcomb (1990) 
recommended that “teachers of agriculture should further develop a curriculum which 
appropriately challenges students at all levels of cognition” (p. 75). 

Not only will a cognitively challenging curriculum enhance cognitive thinking 
levels in students, but the way in which the curriculum is taught will make the difference 
(Whittington and Bowman, 1994). Underbakke, et al. (1993) suggested that the teacher is 
the most powerful control factor that influences students’ development of higher order 
thinking skills. Cano and Martinez (1991) concluded that “agricultural educators [need] 
to challenge students to develop cognitive abilities and critical thinking skills at higher 
levels via the instruction they provide” (p. 28). 



An earlier study of strategic teaching methods concurred with recent research 
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and should consider teaching topics which are meaningful, applicable, and useful in 
students’ lives. Ogle stressed the importance of feedback between students and teachers, 
and also suggested using application and integration activities following the lesson in 
order to promote a deeper understanding of the subject. 



Purpose and Objectives 

The purpose of this study was to assess and compare the cognitive levels of 
instruction among professors from the Pennsylvania Governor’s School for the 
Agricultural Sciences (PGSAS), and the cognitive levels of thought among PGSAS 
scholars. Specifically, the research questions which guided this study were: 

• At what level of cognition were professors actually teaching? 
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• At what level of cognition were students actually operating? 

Methods/Procedures 

Professors 



Population and Sample 

The target population in this study was 16 professors from the PGSAS at The 
Pennsylvania State University. Four professors’ classes were randomly selected for 
analysis. 

Instrumentation 



In 1968, Webb used Bloom’s Taxonomy to create the FTCB to assess the 
cognitive level of classroom discourse (the formal speech or conversation delivered 
during class) professors use when they teach. The FTCB utilizes 55 observable behaviors 
indicative of the various cognitive levels identified by Bloom’s Taxonomy . In the 
“knowledge” category, 17 observable behaviors are listed on the instrument; for 
“comprehension,” 12 observable behaviors are listed; for “application,” four observable 
behaviors are listed; for “analysis,” 11 observable behaviors are listed; for “synthesis,” 
nine observable behaviors are listed; and for “evaluation,” two observable behaviors are 
listed. 



Validity for this instrument was based upon its direct development from Bloom’s 
Taxonomy and the support generally given to this hierarchy of cognitive behaviors. 
Reliability for this instrument was established by coding audio tapes of lectures and 
establishing Spearman Rho reliability coefficients. Inter-rater reliability was 
approximately r = .97. Intra-rater reliability between previous researchers and the 
researchers in this study was approximately r = .96. 



Data Collection 



While attending each professor s class, the researchers recorded the frequency of 
observable teacher behaviors in six-minute intervals. Examples of observable behaviors 
at each level of Bloom’s hierarchy include: “defines meaning of a term” (knowledge 
level); “shows cause and effect relationship” (comprehension level); “applies previous 
learning to new situations” (application level); “shows interaction or relation of elements” 
(analysis level) “formulates hypothesis” (synthesis level); and “evaluates something from 
evidence” (evaluation level). 



In order to collect data on each professor’s background, teaching skills, and 
knowledge of cognitive levels of teaching, professors completed a questionnaire. Each 
professor was also videotaped during the lecture. 
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Data Analysis 



Data were analyzed using the Statistical Package for the Social Sciences (SPSS, 
1990). Frequency of behaviors observed across all cognitive levels was totaled. Then the 
frequency within each cognitive level was divided by the overall total to acquire 
percentages of classroom discourse at each cognitive level. Cross-tabulations, 
frequencies, and means were calculated. 



Students 



Population and Sample 

The second target population for the study was 64 scholars who attended the 
PGSAS during the summer of 1998. The 64 scholars were previously randomly divided 
into four sections of 16 scholars each for the school’s administrative purposes. The 
researchers were members of section one; therefore, for access to the students, section 
one was utilized. 

Since there was a limited amount of time in which the researchers could interview 
scholars immediately after class, the four days in which there was at least a one-hour 
block of time immediately following class were chosen. Four scholars were randomly 
drawn for each interview date, followed by two alternates for each date. 

Instrumentation 



A questionnaire designed by the researchers provided insight into potential 
reactions of the scholar to being interviewed, classes previously taken that would give 
background in the material being taught, and information about the scholars’ interests and 
reasons for attending PGSAS. Scholars completed the questionnaire prior to the 
interview. 



Data Collection 



To understand how students are responding to teachers who teach to higher 
cognitive levels, researchers used think-aloud protocols (verbalization of thought 
processes). Verbal reports were long used as data for psychological research because they 
provided information as to “what is going on in the mind” (Bowen, 1994). Piaget 
validated the use of verbal reports when he used them to test hypotheses based on 
subjects’ responses (Abraham and Renner, 1986). Successful use of think-aloud 
protocols for verbal reports to collect and analyze human thoughts has been legitimized 
by psychologists and intelligence researchers. 



In this study, subjects knew prior to class that they would be interviewed about 
their thoughts during class. The subjects were told the objectives of the study. 
Immediately following class, students were given a hand-held tape recorder and asked to 
watch the videotaped lecture, listen, and audibly recall and describe their thoughts during 
class. 
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Data Analysis for Students 



The audiotapes of cognitive processes were transcribed by a staff assistant. 
Thoughts of students were sorted into six research-generated categories and then 
classified into Bloom’s cognitive levels. The researchers categorized the thoughts as: 

• Thoughts or observations about the professor 

• Nonsense or unrelated thoughts 

• Thoughts connected to previous learning 

• Thoughts about past experiences prompted by class subject matter 

• Deeper leaming/questioning thoughts 

• Thoughts about behavior that got/maintained attention 

Findings and Results 

Assessment of Cognitive Levels Reached by Professors 

Professors in this study taught 43% (see Table 2) of the time at the knowledge 
(compilation of first three categories) level of cognition (range = 8-18%), 30% at the 
comprehension (translation and interpretation) level, (range = 10-20%), 7% at the 
application level, (range = 4-9%), 7% at the analysis level, (range = 6-8%), 7% at the 
synthesis level, (range = 4-10%), and 6% at the evaluation level, (range = 2-12%). The 
most frequently utilized classroom discourse was at the “knowledge of specifics” level. 
The least frequently utilized classroom discourse was at the “application, analysis, 
synthesis, and evaluation” levels. 

The most common type of thought expressed by students (68%, see Table 3) was 
“random or nonsense thoughts” (metacognitive processes unrelated to class subject- 
matter). An example was, “It makes me mad when I can’t find a parking place.” The 
second most common category of thought (12%) was “thoughts about past experiences 
prompted by class subject-matter.” An example was processed while the professor was 
discussing the way pork is currently being bred for leanness. The student thought, “It 
doesn’t matter how lean they make pork, I still won’t like it.” The least used category of 
thought was “deeper leaming/questioning thoughts” (3%). An example was, “If they can 
put windows into cows’ stomachs to measure nutrient absorption, what can we leam to 
help people” ? 



Table 2 

Assessed Level of Cognitive Instruction 
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Level of cognition 


Range (%) 


Total (%) 


Range (f 


1.0 Knowledge of specifics 


8 - 18 


17 


45 - 81 


1 .2 Knowledge of ways and 
means of dealing with 


5 - 12 


11 


27 - 63 


specifics 

1.3 Knowledge of universals 
and abstracts 


4- 18 


15 


29-84 


2.0 Translation 


10-18 


14 


43 - 63 


3.0 Interpretation 


11-20 


16 


48 - 70 


4.0 Application 


4-9 


7 


11 -38 


5.0 Analysis 


6-8 


7 


18-36 


6.0 Synthesis (Creativity) 


4- 10 


7 


16-42 


7.0 Evaluation 


2- 12 


6 


7 - 63 



Note . 1.0+1. 2+1. 3 = Bloom’s “knowledge” level; 2. 0+3.0 = Bloom’s “comprehension” 
level. 



Table 3 

Students’ Categories of Thoughts 



Categories of thoughts 


Range 


% of Time 


Thoughts or observations about the professors 


1 - 5 


3 


Nonsense or unrelated thoughts 


37-86 


68 


Thoughts connected to previous learning 
Thoughts about past experiences prompted by class 


6-20 


10 


subject matter 


2-25 


12 


Deeper leaming/questioning thoughts 


2-7 


3 


Thoughts about behavior that got/maintained attention 


0-6 


4 



Students’ Cognitive Level of Thoughts 

The most common cognitive level of students’ thoughts in class was “knowledge 
ievei” (12.7%, see Tabie 4). Knowledge was considered in two different forms: a) 
searching for, and b) expressing the recognition of basic knowledge. For instance, when 
the professor was showing students the uterus of a pig, one student thought, “Which way 
do they come out [when they are bom]”? This example is a search for knowledge. 
However, when the professor was discussing the domestication of different plant crops, 
and the student thought, “When I saw that blueberries were domesticated in North 
America, I remembered other fruits that were domesticated in the U. S,” the student was 
demonstrating an expression of basic knowledge. 



Table 4 

Comparison of Professors’ and Students’ Cognitive Level of Thought During Class 



Cognitive level 


Professors (%) 


Students (%) 


Knowledge 


43 


12.7 


Comprehension 


30 


11.8 


Application 


7 


2.0 


Analysis 


7 


4.5 


Synthesis 


7 


<1 


Evaluation 


6 


1 


Other 


0 


68 


Total 


100 


100 



The next most used level of cognitive thinking was comprehension (1 1.8%). 
Comprehension involves two forms: a) to understand information, and b) to question the 
information given. For instance, with regard to understanding information, a professor 
was describing the antibodies in a human mother's milk, and the student thought, "If the 
mother is malnourished, the children will be [malnourished] as well because there won't 
be enough nutrients in the milk." The questioning form of comprehension is shown in the 
following situation: A professor was examining the uterus of a pregnant cow, and there 
were no ovaries attached to the uterus. The student thought “Why are the ovaries 
missing? If they were missing her whole life, could she reproduce artificially”? 

The application level of thought involved an average of 2% of the students’ 
thoughts in class. For example, while the professor was discussing the effects of obesity 
on cancer rates, a subject thought, “My mom is a health nut, so she’ll have a better chance 
of not getting [cancer].” 

The analysis level of cognition consumed an average of 4.5% of the thoughts in 
class. For example, when discussing evolution in class, a student was reminded of 
learning about evolution and creation, and wondered, “What are the components of 
evolution and creation that can be combined”? 



Less than 1% of students’ thoughts could be classified at the synthesis level. An 
example of synthesis level thinking occurred during a class discussion on the differences 
between breast milk and formula. A student thought, “Why can’t we make [breast milk 
and formula] the same? I don’t know all the different hormones [contained] in milk. If I 
know the natural ingredients how could I combine them to make perfect synthetic breast 
milk”? 



In this study an average of 1% of students’ thoughts in class were devoted to the 
evaluation level. Over two-thirds (68%) of thoughts generated by students during class 
were “random nonsense thoughts”; these were not classified as part of the cognitive 
assessment. 



Conclusions 

Professors in this study were generally teaching at lower cognitive levels. For 
example, the most common teaching behaviors recorded among professors in this study 
were: basic elicitation of facts, verbalizing from and/or creating graphic representations, 
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making generalizations about concepts or ideas, summarizing and concluding from what 
had been said, and giving reasons for facts. When professors did teach at higher 
cognitive levels, the most common behaviors were: producing unique communication 
and/or divergent ideas, showing the interaction and relationship among elements, and 
applying abstract knowledge in a practical situation. 

Students, on the other hand, primarily thought “random nonsense thoughts” 
during lectures. They rarely thought at the higher cognitive levels no matter the cognitive 
level at which the professor taught. 



Based on the conclusions of this study, the researchers recommend that 
professors: 

© be made aware of cognitive levels of teaching by participating in faculty 
seminars and workshops, or by reading mailings sent to faculty. 

• make students aware of the objectives of the lesson prior to the start of the 
lesson (Perrone, 1994). 

• teach subject matter by linking knowledge with real-life situations and issues 
(Perrone, 1994). 

• provide the opportunity for students’ feedback of their performance relative to 
the goals of their instruction (Perkins & Blythe, 1994). 

• discuss hypotheses within the body of the lecture and assist students in the 
formation of new hypotheses (Bloom et al., 1956). 

• use more visual aids along with “user-friendly” terminology wherever possible 
to attract and maintain the attention and focus of students. 

Based on the conclusions of this study the researchers recommend that students: 

• discipline themselves to pay attention and focus on the materials presented. 

• challenge themselves to think in-class about applications of classroom material 
to their everyday lives. 

• analyze information as it relates to previous and future life situations. 

© synthesize content to follow-through a problem and formulate new hypotheses. 

• evaluate subject-matter to determine effectiveness of decision making in 
possible future situations in their lives. 



Getting students to focus on the topic seems to be the first task. Professors need to 
make known the goals of the lesson before the lesson begins (Ericksen, 1984). Too often 
students do not connect the knowledge of the lesson with the reason for learning it until 
after the lesson has begun and at the time, without a frame of reference, they may not 
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synthesize or evaluate this newfound information. The higher levels of cognition, which 
result from making the connection between new knowledge and intention, needs to take 
place throughout the lecture, so that every new piece of knowledge gained is synthesized 
into the “big picture” being formulated in the student’s mind. 

Students should receive feedback from professors throughout the duration of the 
class, so that students may then use the feedback to assess themselves (Terenzini et al., 
1995). Self-assessment breeds a higher level of understanding of the material taught and 
of one’s own cognitive processes, which theoretically leads students to perform at higher 
levels of cognition including analysis, synthesis, and evaluation. 

Most importantly, however, is the clear need for professors to become aware of 
cognitive levels of teaching (Whittington, 1995). When professors are aware of cognitive 
levels of teaching, they will become aware of those classroom behaviors or teaching 
techniques which help students to think at higher cognitive levels. They will, therefore, 
be able to assess their own teaching, plan their lesson, and revise their classroom 
behaviors in order to teach at higher cognitive levels. 

Students 



An interest in the subject material of the class played a large role in whether or not 
the student maintained attention (Ericksen, 1984). Although a plethora of thoughts were 
present during class, few were related to the material being presented. A large number of 
rhetorical questions were asked during class time. Information that could be applied to 
life or associated with recent circumstances, and was therefore easily understood, was 
more readily absorbed by students. 



Certain teacher behaviors stimulate students’ thought processes. When professors 
asked for input and ideas during class, students reported being more actively engaged in 
the learning process; when there was interaction with the professor, there was more 
motivation to pay attention and participate. Students were also motivated by visual 
stimulation. For example, when researchers asked students what they were thinking 
during a given point in class, they may not recall it until a visual aid from class was 



a! a r’ in fV*r*nt ri F tVi ■ 



at that mnmpnf etnr1<=»nte r^rall Mi/Viat tVi£» nrn fp»eer»r wac 

*■ ■*** rn m. m ^ ‘ VU V V/ V4-JLV4 VV ¥ V ( ( ^4 V V^JLV V A.VUU V> J|_ * • 






discussing and what they were thinking. Students were able to describe various gestures 
professors had used in reference to the subject matter. They stated that when the 
professor asked the class to figure out problems for themselves, more thought processes 
were engaged. 
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